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This study examined the effectiveness of two methods of increasing student learning from 
posterior probability lessons: diagrams and questioning while reading. Undergraduate students 
(N = 245) read a lesson in one of three diagram conditions and one of three questioning-while- 
reading conditions (embedded questions, elaboration interrogations, and control in which they 
read twice). Diagrams helped students in the control group accurately solve posterior 
probability problems on the posttest, but only if the information presented in the diagram was 
also presented in the text. Contrary to expectation, students in the control condition accurately 
solved more posterior probability problems on the posttest than did students in the questioning 
conditions. Poor answer quality for the embedded questions and elaborative interrogations is a 
possible explanation for the unexpected results. 
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Probabilistic reasoning can be difficult for undergraduate students. One type of probabilistic 
reasoning, calculating posterior probability, is particularly challenging (Kahneman & Tversky, 
1972). Calculating posterior probability involves considering both the probability of a particular 
characteristic or disease in a given population (i.e., the base rate) and the probability that the 
identification of that characteristic or disease is accurate (i.e., the specificity). Students 
frequently focus on the specificity information because they consider it more salient than the 
base rate (Kahneman & Tversky, 1972; Sedlemeier & Gigerenzer, 2001). Therefore, it can be 
challenging for students to understand how to calculate posterior probability. However, it is 
important to teach students about probabilistic reasoning, including skills such as accurately 
calculating posterior probability, so that they can be better consumers and citizens in modern 
society (Garfield & Ben-Zvi, 2008). The present study examined the effects of two factors that 
may promote student learning from a lesson on calculating posterior probability: diagrams and 
questioning while reading. 

Diagrams 

One technique that may foster students’ learning is including relevant visual representations. 
According to the multimedia principle, learning is improved when texts are accompanied by 
relevant visuals (Mayer, 2009), because students develop a verbal mental model from the text 
and a visual mental model from the visuals. Students are more likely to make connections 
among different ideas when they have the information represented in two mental models. These 
connections prompt deeper processing of the text and increased learning. 

At present, it is unclear whether learning is improved when information presented in a 
diagram is also presented in text. According to the redundancy principle, information presented 
in the diagram should not also be presented in the text (Sweller, Ayres, & Kalyuga, 2011), 
because presenting information in both text and diagram increases the amount of information the 
student has to process, which may diminish learning (Mayer, 2009). However, the redundancy 
principle is based on studies using causal diagrams for scientific concepts (e.g., Chandler & 
Sweller, 1991). It has not been explored with numeric information in math lessons. It is possible 
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that having the numeric information presented in both diagram and text could facilitate 
connections between the verbal and visual information. Students may be better able to connect 
the concepts presented in the text and diagram if the information presented in the diagram is also 
presented in the text. Therefore, it is uncertain if including the information in the diagram as 
well as in the text will improve student learning about posterior probability. 

Questioning while Reading 

A second technique that may improve student learning is to have the students answer 
questions about the lesson while reading. Two types of questions are commonly used: embedded 
questions and elaborative interrogations. Embedded questions are simple questions about 
specific ideas stated in the lesson. Embedded questions are useful for directing students’ 
attention to important information. Answering embedded questions has been found to increase 
learning from text; however, these benefits may be limited to certain student populations (e.g., 
less-skilled readers, Callendar & McDaniel, 2007). Elaborative interrogations are typically 
“why” questions intended to prompt the student to integrate an idea from the lesson with other 
ideas from the lesson or with background knowledge (e.g., McDaniel & Donnelly, 1996; 
Pressley, Symons, McDaniel, Snyder, & Turnure, 1988). Elaborative interrogations are intended 
to stimulate deeper processing of the lesson, thereby increasing learning (e.g., Ozgungor & 
Guthrie, 2004). Elaborative interrogation has been found to improve learning across a variety of 
student populations (cf. Dunlosky, Rawson, Marsh, Nathan, & Willingham, 2013). Given the 
previous literature on both embedded questioning and elaborative interrogation, it is likely that 
including these types of questions will improve student learning from a lesson on posterior 
probability. 

The use of questioning while reading may be particularly beneficial when visual 
representations are included in the lesson. Students do not always attend to the visual 
representations in text (Schiller, Scheiter, Rummer, & Gerjets, 2012). Embedded questioning 
may be useful for directing students’ attention towards the visual representations, particularly if 
the questions are about the visual representations. In addition, integrating visual and verbal 
information typically promotes learning (Mayer, 2009). Elaborative interrogation may be useful 
for guiding students to integrate the visual and verbal information. This can be accomplished by 
asking elaborative interrogations that encourage the students to use information in both the 
diagram and the text. 

The present study examined the effects of diagrams and questioning on learning about 
posterior probability calculations. Given the previous literature on visual representations (e.g., 
Mayer, 2009), we predict that students will learn more if their assigned lesson includes a 
diagram. However, it is uncertain whether students will learn more if the information presented 
in the diagram is also presented in the text. In addition, we predict that students in the two 
questioning-while-reading conditions will learn more than students in the control condition, who 
will simply read the lesson twice. Finally, we predict that students who receive both diagrams 
and questions will learn more than students who receive only one or the other, or neither. 


Methods 
Participants were 248 undergraduates at a large, upper Midwestern university who earned 
extra credit in their introductory psychology course for participation. Three participants left the 
study before completing the posttest; their data were not included. Of the remaining 245 
participants, there were 158 females and 86 males (one participant did not report gender) with an 
average reported age of 19.05 years (SD = 2.89 years). English was reported as the native 
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language of 164 students; 80 students reported a native language other than English (one student 
did not report native language). 
Materials 

The lesson was adapted from a textbook by Heuer (1999) and included materials from 
Sedlmeier and Gigerenzer’s (2001) experiments. There were two examples of scenarios in 
which base rate and specificity probabilities were presented, and then the posterior probability 
was calculated using natural frequencies. For the diagram conditions, there was a diagram for 
each of the two examples. For the diagram without redundant text condition, the information in 
the diagram was not also presented in the text. For the diagram with redundant text condition, 
the information in the diagram was also presented in the text (see Table 1). 


Table 1: Lesson Excerpts from Diagram without Redundant Text Condition and 
Diagram with Redundant Text Condition 


Diagram without redundant text Diagram with redundant text 


Imagine 100 cases in which the pilot has a similar encounter. 
Based on the base rate, we know that 85 of these encounters will 
be with Vietnamese fighter jets, 15 with Cambodian. 


Imagine 100 cases in which the pilot has a similar encounter. 
Based on the base rate, we know how many of these encounters 
will be with Vietnamese fighter jets, and how many with 


Cambodian. Based on the specific case information, we know that 
80 percent Vietnamese fighter jets will be correctly identified as 
Vietnamese, while 20 percent will be incorrectly identified as 
Cambodian. Similarly, 80 percent Cambodian fighter jets will be 
correctly identified as Cambodian, while 20 percent will be 
incorrectly identified as Vietnamese. This is shown in the diagram 
below. 


100 aircraft 


a 


85 Vietnamese 15 Cambodian 


a ™ Aa ™ 


68 Identified as 17 Identified as 12 Identified as 3 Identified as 
Vietnamese Cambodian Cambodian Vietnamese 


This makes a total of 71 Vietnamese and 29 Cambodian sightings, 
but not all of these sightings were correct. Therefore, when the 
pilot claims the attack was by a Cambodian fighter jet, the 
probability that the fighter jet was actually Cambodian is only 
12/29ths or 41 percent. 


Based on the specific case information, we know that 80 percent or 
68 of the 85 Vietnamese fighter jets will be correctly identified as 
Vietnamese, while 20 percent or 17 will be incorrectly identified as 
Cambodian. Similarly, 80 percent or 12 of the 15 Cambodian 
fighter jets will be correctly identified as Cambodian, while 20 
percent or 3 will be incorrectly identified as Vietnamese. This is 
shown in the diagram below. 


100 aircraft 


a 


85S Vietnamese 15 Cambodian 


aA ™ aA ™ 


68 Identified as 17 Identified as 12 Identified as 3 Identified as 
Vietnamese Cambodian Cambodian Vietnamese 


This makes a total of 71 Vietnamese and 29 Cambodian sightings, 
of which only 12 of the 29 Cambodian sightings are correct; the 
other 17 are incorrect sightings of Vietnamese fighter jets. 
Therefore, when the pilot claims the attack was by a Cambodian 
fighter jet, the probability that the fighter jet was actually 
Cambodian is only 12/29ths or 41 percent. 


For the embedded questioning and elaborative interrogation conditions, there was one question 
after approximately every paragraph. For diagram and questioning conditions, a question was 

asked after each of the diagrams. For the diagram without redundant text condition, answering 
the question correctly required using information from the diagram. However, for the diagram 


with redundant text condition, the question could be answered using information from the text or 
the diagram (or both). Embedded questions were simple (e.g., “How many fighter jets identified 
as Vietnamese are actually Cambodian?”) and their answers were explicitly stated in the text 
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and/or clearly presented in the diagram. Elaborative interrogation questions were complex 
“how” and “why” questions (e.g., “How are the base rate and the specific case information used 
to calculate the probability the fighter jet is Cambodian?”). The answers to elaborative 
interrogations were implicit in the text; the participant had to connect different ideas presented in 
the text and/or diagrams. All lessons began with the reminder that the participant would be 
asked to answer questions based on the information in the lesson. The lesson in the control 
condition also began with the reminder that participants needed to read the lesson twice. At the 
end of the control condition lesson, participants were reminded that if they had only read the 
lesson once, they needed to go back to the beginning and re-read. 

Measures 

Pre/posttest. Prior to reading the lesson, participants were asked to solve the following 
posterior probability problem: “All medical tests have error. For example, the serum test 
screens pregnant women for fetuses with Down syndrome. The test is a very good one, but not 
perfect. Roughly 100 fetuses out of 10,000 have Down syndrome. Of these 100 fetuses with 
Down syndrome, 90 pregnant women will have a positive test result. Of the remaining 9,900 
unaffected fetuses, 99 pregnant women will still have a positive test result. What is the 
probability a pregnant woman who has a positive result on the test actually has a fetus with 
Down syndrome?” The problem was presented only as text; no visuals were included. The 
posttest consisted of four posterior probability problems similar to the pretest. The problems 
were presented as text only; no visuals were included. 

Procedure 

Participants were randomly assigned to conditions. Participants engaged in the experimental 
tasks individually in small groups of 2 to 6. All participants in a particular group were in the 
same condition. All tasks were paper-and-pencil (i.e., they were not administered on a 
computer). After providing informed consent, participants were given a posterior probability 
problem to solve as a pretest. Then, they were given the lesson and an answer sheet for their 
questions, if they were in one of the questioning conditions. The experimenter instructed them to 
read the lesson carefully because they would be asked to answer questions and solve problems 
based on its information. Participants in the control condition were instructed to read the lesson 
twice. Participants in the questioning condition were instructed to answer the numbered 
questions in the lessons when they came to them, in the appropriate places on the answer sheet. 
When participants finished reading the lesson, they completed a distractor task of 21 single-digit 
multiplication and division problems. Then, they completed the posttest. Finally, they self- 
reported their demographic information. Experimental tasks were completed in a single session, 
which were approximately 45 minutes in length. 

Scoring 

Pre- and posttests. Pre- and posttests were scored for accuracy. The number of accurately 
solved problems was the score for the pretest and posttest. The maximum posttest score was 4. 

Answers to embedded questions. Answers to embedded questions were scored 
dichotomously as correct or incorrect. Correct answers were scored as ‘1’ and incorrect answers 
as ‘0.’ 

Answers to elaborative interrogations. Because the elaborative interrogations were more 
complex than the embedded questions, their answers were scored using a rubric instead of 
dichotomously. The rubric had the following categories: ideal, adequate, inadequate, circular, 
wrong, and missing. Elaborative interrogations were designed to prompt integration of concepts 
within the lesson. Therefore, an answer was considered ideal if it correctly included three or 
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more concepts presented in the lesson. Adequate answers correctly included two concepts. 
Inadequate answers included only one concept. Circular answers repeated the information in the 
question (e.g., EI: “Why is the base rate important when calculating probability?” Circular 
Answer: “Because it is important information you need to use to calculate probability.”). Wrong 
answers provided incorrect or inappropriate information. Missing answers were either responses 
such as “I have no idea” or the complete absence of answers. An ideal answer was scored as ‘3.’ 
An adequate answer was scored as ‘2.’ An inadequate answer was scored as ‘1.’ A missing, 
incorrect, or circular answer was scored as ‘0.’ The scores for all 6 of the answers to the 
elaborative interrogations were summed for a measure of answer quality. 


Results 

We first examined the effects of the diagram and questioning conditions on the posttest 
scores. We separated the participants based on whether they answered the pretest correctly (N = 
47) or incorrectly (V = 198). An ANOVA was conducted with diagram condition and 
questioning condition as fixed factors and posttest scores as the dependent variable. Bonferroni 
corrections were used for multiple comparisons. Figure | presents the data for each of the pretest 
accuracy groups by condition. 

For participants who answered the pretest correctly, diagrams had a positive effect on 
posttest scores F(2, 45) = 8.26, p= .001. There was a benefit of diagrams both without and with 
redundant text compared to the text-only condition, (28) = 3.04, p = .01,Cohen’s d= 1.01; #28) 
= 3.98, p = .001, Cohen’s d = 1.26, respectively. There was no difference between the two 
diagram conditions. In addition, there was no effect of questioning. 

For participants who answered the pretest incorrectly, there was no overall effect of diagrams 
on posttest performance. However, there was an interaction between diagram condition and 
questioning condition, F(2, 196) = 3.04, p = .02, such that diagram condition affected 
performance in the read-twice control condition, t(61) = 2.56, p = .04, Cohen’s d= .63, but not 
the other questioning conditions. In the read-twice control condition, participants in the diagram 
with redundant text condition had higher scores than did participants in the text-only condition. 
There were no differences between the diagram without redundant text condition and the other 
diagram conditions. For participants who answered the pretest incorrectly, there was also a main 
effect of questioning condition, F(2, 196) = 13.56, p < .001. Unexpectedly, participants in the 
control condition had higher posttest scores than did participants in the embedded questioning 
condition, t(118) = 4.49, p < .001, Cohen’s d = .74, and participants in the elaborative 
interrogation condition, t(154) = 4.34, p < .001, Cohen’s d= .69. 


Pretest Correct Pretest Incorrect 


Bret vy Bitex cry 
Cragam Cregram 
gino gytnouwt 
* Redundant = Redundant 
3 Text 3 Text 
= | | Ciegam wen 4 Ckagramn weh 
£ | UiResundant & CiRedundant 
ma Text 3 Text 
2 a 
a 
Contra! Embedded Elaborative Contro Embedded Elaborative 


Questioning interrogation o. orig Interrogaticn 


Figure 1: Post-Test Scores by Diagram and Questioning Conditions 
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The results indicate that diagrams were helpful for participants who initially did well on the 
posterior probability problem at pretest, aiding them to continue to do well on similar problems 
at posttest. In contrast, participants who did not understand how to answer the pretest problem 
benefited from diagrams when they read the lesson twice, but not when they answered questions 
while reading. Moreover, diagrams appear to be helpful only if they were accompanied by 
redundant text for participants who answered the pretest problem incorrectly. Contrary to 
expectations, questioning while reading was not beneficial for learning how to solve posterior 
probability problems. Indeed, reading twice appeared to yield higher scores on the posttest than 
either elaborative interrogation or embedded questioning for participants who answered the 
pretest problem incorrectly. 

Given the unexpected finding that embedded questioning and elaborative interrogation 
conditions were not helpful for learning, we examined the quality of the answers to the questions 
in the embedded questioning and elaborative interrogation conditions (see Scoring section for 
scoring of answers). Because of the different patterns of findings based on pretest accuracy, we 
tested for differences in answer quality based on pretest accuracy. Then, we examined 
associations between answer quality and learning. 

For the embedded questioning condition, participants answered an average of 3.08 questions 
correctly (SD = 1.70) out of 6 questions total. We conducted a one-way ANOVA with accuracy 
of answers to the embedded questions as the dependent variable and pretest accuracy as the 
independent variable. There were no differences in answer quality between participants who 
answered the pretest correctly (M = 3.00, SD = 1.70) and participants who answered the pretest 
incorrectly (M = 3.11, SD = 1.72). We used Pearson product-moment correlations to test for 
associations between accuracy of answers to the embedded questions and posttest scores. There 
were no associations between answer quality and posttest scores for either participants who 
answered the pretest correctly or participants who answered the pretest incorrectly. 

For the elaborative interrogation condition, participants received an average score of 4.80 
(SD = 2.27), out of a maximum possible score of 18. Approximately 37% of the answers were 
missing, circular, or wrong, 51% of the answers were inadequate, 9% of the answers were 
adequate, and 4% were ideal. We conducted the same statistical tests with the answers to the 
elaborative interrogations as we did with the answers to the embedded questions. As with 
embedded questioning, there were no differences in answer quality between the participants who 
answered the pretest correctly (M = 4.82, SD = 2.39) and participants who answered the pretest 
incorrectly (M = 4.69, SD = 1.70). Also similar to embedded questioning, there was no 
association between the answer quality for the elaborative interrogations and posttest scores for 
participants who answered the pretest correctly. However, for participants who answered the 
pretest incorrectly, there was a positive correlation between the quality of the answers to the 
elaborative interrogations and posttest scores, r= .39, p = .002. 


Discussion 

The purpose of this study was to examine the effects of diagrams and questioning while 
reading on learning from a lesson on calculating posterior probability. We had hypothesized that 
including diagrams in the lesson would improve learning, but were uncertain whether including 
text redundant with the information in the diagram would affect learning. For students who 
answered the pretest problem correctly, diagrams improved learning, whether or not they were 
accompanied with redundant text. However, for students who answered the pretest problem 
incorrectly, there was a benefit of diagram with redundant text, relative to text only, but only for 
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the control (i.e., read twice) condition. There was no reliable benefit of diagram without 
redundant text. These findings are consistent with the multimedia principle, which holds that 
including relevant visuals in lessons improves learning (Mayer, 2009). 

However, our findings are inconsistent with the redundancy principle, which states that 
information should be presented in the text or diagram, but not both (Sweller et al., 2011). For 
students who solved the pretest problem incorrectly, there was no effect of redundant text. But, 
for students who solved the pretest incorrectly in the read-twice condition, there was a reliable 
benefit of diagrams only if they were accompanied by redundant text. In other words, presenting 
the information in a diagram instead of the text did not appear to be consistently helpful. For 
some students, the information needed to be in the diagram as well as the text for a benefit to be 
observed. It may be that presenting the information in both the text and diagram helped the 
students connect the two representations. The integration of the information may have led to 
more in-depth learning, resulting in better comprehension and application of the lesson. 

We had hypothesized that answering questions while reading would have benefited learning. 
However, there was no effect of questioning for students who answered the pretest problem 
correctly. Moreover, students who answered the pretest problem incorrectly had higher posttest 
scores when they read twice as opposed to answer questions while reading. One possible 
explanation for the unexpected lack of benefit of questioning is that the quality of the answers to 
both the embedded questions and elaborative interrogations was generally quite poor. According 
to Jiang and Elen (2011), answering questions while reading only improves learning from text if 
students answer the questions as they were intended. On average, students answered only about 
half of the embedded questions correctly. This is surprising given that the answers to the 
questions were explicitly stated in the text; however, the undergraduate participants may have 
been confused by the simplicity of the questions presented to them. This confusion could have 
led to inaccurate answers to the embedded questions and diminished performance on the posttest 
for students who answered the pretest problem incorrectly. For the elaborative interrogations, 
students did not provide answers indicative of the deep processing these questions were intended 
to stimulate. The overwhelming majority of the answers to the elaborative interrogations were 
missing, circular, wrong, or inadequate. 

Given our findings regarding answer quality, it is not surprising that both questioning 
conditions were detrimental, rather than beneficial, to learning for students who answered the 
pretest incorrectly. They were likely distracted by the questions and thereby benefited only if 
they read twice. However, students who answered the pretest problem correctly were not 
affected by questioning even though their answer quality was similar to those who answered the 
pretest problem incorrectly. Students who answered the pretest problem correctly may have 
been better able to focus on lesson content and suppress the distraction of the questions. 

The results indicate that better answers to elaborative interrogations were associated with 
better learning from the lesson for students who lack prior understanding of the content. 
Therefore, benefits of elaborative interrogation might be observed if answer quality was stronger. 
We propose two ideas to improve answer quality in future studies. The first is practice training 
in which students receive examples of what types of answers are expected of them. This practice 
training would provide a model for the students, which could improve their answer quality (cf. 
Dornisch, Sperling, & Zeruth, 2011, for discussion). The second is to provide students with 
feedback on their answers either in the experiment, through their peers while working 
collaboratively (after training), or through computer-based tutoring. If these ideas could 
effectively improve answer quality, positive effects of elaborative interrogation might be seen. 
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Posterior probability, like many probabilistic reasoning concepts, is frequently challenging 
for students to learn. In this study, we sought to improve student learning on posterior 
probability problems by incorporating diagrams and questioning while reading into lessons. Our 
finding that diagrams assisted learning is consistent with the multimedia principle (Mayer, 2009). 
However, our finding that redundant text has a neutral or positive effect on learning, depending 
on the population, is inconsistent with the redundancy principle (Sweller et al., 2011). Our 
findings regarding questioning are inconsistent with our expectations based on previous 
literature. We believe that poor answer quality is the reason for the unexpected results, and we 
suggest some methods of improving answer quality. This work provides some guidance about 
how to improve student learning about probabilistic reasoning, while also highlighting the 
challenges inherent in this complex domain. 
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